-
Notifications
You must be signed in to change notification settings - Fork 839
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix ConcurrentModificationException #6732 #6746
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #6746 +/- ##
============================================
- Coverage 90.08% 90.06% -0.02%
+ Complexity 6464 6462 -2
============================================
Files 718 718
Lines 19528 19530 +2
Branches 1923 1923
============================================
- Hits 17591 17590 -1
- Misses 1348 1350 +2
- Partials 589 590 +1 ☔ View full report in Codecov by Sentry. |
api/all/src/main/java/io/opentelemetry/api/internal/ConfigUtil.java
Outdated
Show resolved
Hide resolved
api/all/src/main/java/io/opentelemetry/api/internal/ConfigUtil.java
Outdated
Show resolved
Hide resolved
System.getProperties().entrySet().stream() | ||
.filter(entry -> normalizedKey.equals(normalizePropertyKey(entry.getKey().toString()))) | ||
.map(entry -> entry.getValue().toString()) | ||
System.getProperties().stringPropertyNames().stream() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
stringPropertyNames
calls entrySet
without extra synchronization so ConcurrentModificationException
should still be possible
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
stringPropertyNames
promises an unmodifiable Set<String>
based on enumerateStringProperties()
, which at least in JDK8 seems to be synchronized.
Any other idea then? The last that comes to my mind is using clone()
and do the operation on the copy. I'm all ears!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In jdk11 that method is not synchronized.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, super weird.
Was also checking the Android implementation, same thing.
https://cs.android.com/android/platform/superproject/main/+/main:libcore/ojluni/src/main/java/java/util/Properties.java
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does enumerating over System.getProperties().keys()
help?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added commit with clone()
. This did prevent the exception (at least in jdk 8).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clone()
is going to be a lot more expensive
fwiw keys()
appears to be marked as synchronized
: https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/Hashtable.java#L256
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fwiw keys() appears to be marked as synchronized
I don't think that is enough. https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/Hashtable.html says
The iterators returned by the iterator method of the collections returned by all of this class's "collection view methods" are fail-fast: if the Hashtable is structurally modified at any time after the iterator is created, in any way except through the iterator's own remove method, the iterator will throw a ConcurrentModificationException. Thus, in the face of concurrent modification, the iterator fails quickly and cleanly, rather than risking arbitrary, non-deterministic behavior at an undetermined time in the future. The Enumerations returned by Hashtable's keys and elements methods are not fail-fast; if the Hashtable is structurally modified at any time after the enumeration is created then the results of enumerating are undefined.
Similarly to how you can use iterator from synchronized wrapped created with Collections.synchronized...
safely by synchronizing on the collection you could do the same here.
Properties properties = System.getProperties();
synchronized (properties) {
String systemProperty =
properties.entrySet().stream()
.filter(entry -> normalizedKey.equals(normalizePropertyKey(entry.getKey().toString())))
.map(entry -> entry.getValue().toString())
.findFirst()
.orElse(null);
if (systemProperty != null) {
return systemProperty;
}
}
Alternatively you could reduce the chance of race with making a copy of system properties once.
@SuppressWarnings({"NullAway", "NonFinalStaticField"})
private static Map<String, String> systemProperties;
static {
initialize();
}
// Visible for testing
static void initialize() {
systemProperties = System.getProperties().entrySet().stream().collect(
Collectors.toMap(entry -> normalizePropertyKey(entry.getKey().toString()),
entry -> entry.getValue().toString()));
}
public static String getString(String key, String defaultValue) {
String normalizedKey = normalizePropertyKey(key);
String systemProperty = systemProperties.get(normalizedKey);
if (systemProperty != null) {
return systemProperty;
}
...
}
with this you'll lose the ability to pick up modifications to system properties. I doubt that this is actually something we need outside of tests, where you can call initialize
again. Before implementing this approach I'd ask @jack-berg whether this is fine with him.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alternatively you could reduce the chance of race with making a copy of system properties once.
with this you'll lose the ability to pick up modifications to system properties
if we can scope this one time reading of system properties once per SDK creation that seems ideal (so you can set system properties and then instantiate the SDK)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if we can scope this one time reading of system properties once per SDK creation that seems ideal (so you can set system properties and then instantiate the SDK)
so something like:
private static final Properties systemProperties = (Properties) System.getProperties().clone();
??
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how about System.getProperties().stringPropertyNames()
is more light than clone
import java.util.Locale; | ||
import java.util.Map; | ||
import java.util.Objects; | ||
import java.util.Properties; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
import java.util.Locale; | |
import java.util.Map; | |
import java.util.Objects; | |
import java.util.Properties; | |
import java.util.Collections; | |
import java.util.Locale; | |
import java.util.Map; |
((Properties) System.getProperties().clone()) | ||
.stringPropertyNames().stream() | ||
.filter(propertyName -> normalizedKey.equals(normalizePropertyKey(propertyName))) | ||
.map(System::getProperty) | ||
.filter(Objects::nonNull) | ||
.findFirst() | ||
.orElse(null); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
((Properties) System.getProperties().clone()) | |
.stringPropertyNames().stream() | |
.filter(propertyName -> normalizedKey.equals(normalizePropertyKey(propertyName))) | |
.map(System::getProperty) | |
.filter(Objects::nonNull) | |
.findFirst() | |
.orElse(null); | |
String systemProperty = | |
Collections.unmodifiableSet(System.getProperties().stringPropertyNames()).stream() | |
.filter(propertyName -> normalizedKey.equals(normalizePropertyKey(propertyName))) | |
.map(System::getProperty) | |
.findFirst() | |
.orElse(null); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unmodifiable set is superfluous since there's no attempt to modify the contents.
I think we can add a hook method is like this:
origin code can be like this:
unit test may be like this:
|
.map(entry -> entry.getValue().toString()) | ||
.findFirst() | ||
.orElse(null); | ||
((Properties) System.getProperties().clone()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be nice to have a test recreating this CME and demonstrating that this fixes it. However, I spent a decent amount of time trying to hammer this with concurrent reads and writes and was unable to recreate it, so it could be specific to some version of java or the jdk.
I'm content give this a shot even without strong proof that it resolves it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like it does not reproduces on jdk 11+ probably because of https://github.com/openjdk/jdk11u/blob/cee8535a9d3de8558b4b5028d68e397e508bef71/src/java.base/share/classes/java/util/Properties.java#L159-L165
On jdk8 it reproduces with
import java.util.concurrent.Executor;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicInteger;
public class Main {
public static void main(String[] args) {
int threads = 4;
AtomicInteger counter = new AtomicInteger();
Executor executor = Executors.newFixedThreadPool(threads);
for (int i = 0; i < threads; i++) {
executor.execute(new Runnable() {
@Override
public void run() {
while (true) {
String property = "prop " + counter.getAndIncrement();
System.setProperty(property, "a");
System.getProperties().remove(property);
}
}
});
}
while (true) {
System.getProperties().entrySet().stream()
.filter(
entry -> "x".equals(entry.getKey().toString())
)
.map(entry -> entry.getValue().toString())
.findFirst()
.orElse(null);
}
}
}
ConcurrentModificationException
does not reproduce with the proposed fix. It doesn't reproduce with
Properties properties = System.getProperties();
synchronized (properties) {
properties.entrySet().stream()
.filter(
entry -> "x".equals(entry.getKey().toString())
)
.map(entry -> entry.getValue().toString())
.findFirst()
.orElse(null);
}
either. Unlike the proposed fix this version does not make extra copies of system properties.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good find @laurit!
I know there was some skepticism about taking a lock on System.getProperties()
, but it seems ok to me. The work we're doing inside the lock is narrow and its hard for me to imagine how we could end up in a dead lock scenario.
Anything missing here, @jack-berg ? Would be happy to see this in a release soon :). |
Sorry for not getting back on this @neugartf. We talked about this in the 10/10/2024 java SIG, just before the 1.43.0 release. Decided it needed more work:
So in summary, to get this back on track, we need:
|
Thanks for getting back (and with that detail)! I'll invest some time to get this reproduced on the test app there ( |
That would be great. If you can just show that you can read and write to |
Hey @jack-berg, was able to reproduce the crash with this change. The fix with |
Thanks for doing that research @neugartf. I have a slight preference to go with @laurit's synchronized approach, because I have a general aversion to implementing What do you think about adding a method for accessing a safe copy of |
Merged #6841 so closing this. |
Didn't had much time recently, but looks good! Thanks for taking care of that :). |
Could be nicer with jdk > 8, but that's how it is :). I added a filter to make sure only string key/values from
Properties
are read into our logic.